Here Jesus was not warned to flee, but the angel warned Joseph in a dream.
Joseph was warned in a dream to take the young child and his mother and flee to Egypt untill he hears from God again. (Matthew 2.13) So "the holy family" that fled to Egypt, were Joseph and Mary and Jesus.
EgyptMat 2:13 Now when they had departed, behold, an angel of the Lord appeared to Joseph in a dream, saying, "Arise, take the young Child and His mother, flee to Egypt, and stay there until I bring you word; for Herod will seek the young Child to destroy Him."
Actualy Mary and Joseph fled only once to Egypt from king Herod.
An angel came to Joseph in a dream and warned him to take his family to Egypt immediately as Herod was looking for the child jesus.
Mary , Joseph and the baby Jesus fled in the night to Egypt.
God warned Joseph in a dream to take Mary and Jesus to Egypt to avoid the wrath of Herod.
The death of King Ptolemy caused the Auletes to flee Egypt.
Joseph and Mary fled with the baby Jesus to Egypt, as the angel told Joseph to flee as the king Herod wanted to kill all the babies.
Jesus escaped to Egypt from Bethlehem, not Nazareth. King Herod, who wanted to kill him, ordered the massacre of all infant boys in Bethlehem, prompting Joseph and Mary to flee to Egypt with Jesus to keep him safe.
1876
The angel never actually said to go to a town. He merely said to flee to Egypt and to wait for Herod to die.