Caroline frequently attended kindergarten in classes that were organized by her mother, and she was often photographed riding her pony "Macaroni" around the White House grounds. One such photo in a news article inspired singer-songwriter Neil Diamond to write his Top Ten hit song, "Sweet Caroline", which he revealed when he performed it for Caroline's 50th birthday. As a small child, Caroline received numerous gifts from dignitaries, including a puppy from Soviet Premier Nikita Khrushchev and a Yucatán pony from Vice President Lyndon B. Johnson. Historians described Caroline's childhood personality as "a trifle remote and a bit shy at times" yet "remarkably unspoiled. " "She's too young to realize all these luxuries", her paternal grandmother, Rose Kennedy, said of her. "She probably thinks it's natural for children to go off in their own airplanes. But she is with her cousins, and some of them dance and swim better than she. They do not allow her to take special precedence. Little children accept things".
Schlossberg was born in New York City to an Orthodox Jewish family.   His parents Alfred I. Schlossberg and Mae Hirsch were both children of Ukrainian immigrants. Alfred founded a textile-manufacturing business and was president of the Park East Synagogue on New York's Upper East Side , where Ed studied Hebrew and had a Bar Mitzvah .