Yes
It seems like your confusion is arising from the ambiguous case of "one of your [plural x]". Does "one of your own attacks" imply that you have to have multiple attacks in order to use the ability? In English, sometimes it might. For example, if I say, 'You will need to drive one of your own cars today," it is strongly implied that you have multiple cars - if I knew you had only one car, I would say instead "You will need to drive your own car today." On the other hand, if I receive a bulletin from my child's school saying "One of your child's parents must be in attendance at this event," I do not conclude that the school must not have any students with only one parent.
Thus, in English, the case of the expression is ambiguous and could mean "a set with more than one" or "a set of at least one" depending on context. While the PHB rules do not explicitly explain the meaning of this expression, fortunately we are given plenty of examples that make sense only if "a set of at least one" is the meaning.
Consider that if features keyed to "one of your x" could only be used when "your x" was plural, then:
A bard could not use cutting words (54) when they had only one use left of bardic inspiration.
A battle master who took the Commander's Strike maneuver (74) at 3rd level could not use it until they also obtained extra attack at 5th level.
A character with the Lucky feat (167) could not use it after a die roll was made if they were down to their last luck point.
If you had used the animal friendship spell (212) but had only one companion, that companion could harm the charmed animal without ending the spell.
If you used the mirror image spell (260) and your initial three images had been reduced to one, that one final image could never be targeted.
Since all of these interpretations are clearly nonsensical, we can be confident that the Pact of the Chain ability should not be interpreted like them. That is, Pact of the Chain can be used even when the Warlock has only one attack granted by the Attack action.
And, this conclusion makes internal sense as well, for as Eddymage points out in his answer, only Pact of the Blade Warlocks get extra attack. Thus a single class Pact of the Chain Warlock will always have a single attack, only. If having a single attack meant that the warlock could not forgo that attack, then the Pact of the Chain would include a feature unusable by single class Pact of Chain Warlocks, which doesn't make sense either.
Edit: My answer assumed that OP was confused by the "one of your attacks" language. However, as commentary by Daniel R. Collins and OP Maiko Chikyu later established, OP's actual concern was whether you could forgo an attack if taking the Attack action meant you had only one attack - because if the attack was forgone, did you really take the Attack action when you did not make an attack? In that case, this question is actually answered by, and a duplicate of, Can a Battle Master fighter use the Commander's Strike maneuver when they only have one attack?