In this paper, we focus on few-shot aspect-based sentiment analysis (ABSA) and try to solve it with in-context learning (ICL) paradigm. However, the effectiveness of ICL is highly affected by retrieved in-context examples. Previous works generally leverage the semantic similarity between the candidate examples and test input to retrieve examples. However, they may yield sub-optimal results for this task. This is because considering only the overall semantic perspective may leave some useful examples, which have syntactic structural relevance to the test input or share identical sentiments and similar aspects to one unretrievable. To address this shortcoming, we advocate retrieving in-context examples for few-shot ABSA by simultaneously considering three perspectives, overall semantics, syntactic structure relevance, and aspect-sentiment semantics. To achieve this, we construct positive and negative pairs from these three perspectives and train the demonstration retriever using contrastive learning. Experimental results on four ABSA datasets show that our retrieval framework can significantly outperform baselines across the board. Moreover, to understand factors influencing ICL performance on few-shot ABSA, we conduct extensive analysis in various scenarios, which can inspire and advance future research.