כל אהבה שהיא תלויה בדבר
Any love that depends on something,
if that thing is no more,
the love is no more. 
Rabbi Samson Raphael Hirsch wrote:
“Wherever love is rooted in the spiritual and moral worth of the beloved person, there the love will be as abiding as the values on which it is founded. But a love based on physical attraction will not outlast those fleeting charms.” 
Rabbi Joseph Aknin, the Rambam’s disciple, put it quite simply:
“[When a man loves a woman only for her beauty]…If the woman falls sick and loses her beauty, gone is his love for her.” 
Rabbi Isaac Unterman understands “thing” as “a material cause”:
“A corporeal cause is subject to constant change and as soon as this change takes place, the cause vanishes and love evaporates.” 
This seems as if it should be self-evident.
Its mention in Pirkei Avot suggests that even 2000 years ago and more, some “love” was based only on the physical attractiveness of the other. Classical Greek culture extolled beauty. About this, Matthew Arnold is reported to have observed: “The Greeks worshiped the holiness of beauty; the Jews found beauty in holiness.” 
It’s a bit unfair to the classical Greeks, perhaps. I think that their belief might have been more like: the perception of beauty — as uplifting and purifying as it can sometimes be — is a glimpse of Divine Beauty.
American culture has made the worship of mere physical beauty pervasive — leading to many serious problems in the lives of children, teens and adults — both male and female. It has made “beauty” (however defined) synonymous with “self-worth.”
For Jewish tradition, the “holy” is the beautiful:
“Worship G-d in the beauty of holiness.” 
On the other hand, the Jewish ideal of love far from decries sexual enjoyment:
“In Judaism, as important as pru u’rvu (procreation) may be, onah, that is, sexual pleasure as an expression of deep love, is even more important. Note the words of Ramban: ‘Speak words which arouse her to passion, union, love, desire and eros.’ (Epistle of Holiness) Of course, such words and actions should be reciprocated by wife to husband…
A second approach comes to mind. Martin Buber speaks of an I-it encounter, where the ‘I’ relates to the other as a thing, an object to be manipulated and used to satisfy the ‘I.’ This in contrast to the I-thou [or: I-you] encounter where the other is a persona, a subject to be considered and loved.
Hundreds of years before Buber, Rambam in his commentary to the Mishnah (Avot 1:16) wrote about love between husband and wife as empathetic friendship, a camaraderie involving a caring responsiveness, a sharing of innermost feelings…a relationship of emotional rapport rooted in faith and confidence…
One last approach. In many ways love is not only holding on but letting go. To be sure, love involves embracing the other, but in the same breath it allows the other to realize his or her potential. This is the great challenge of harmonization. How can I be one with you while letting you be who you are? On the other hand, how can you be who you are without our becoming distant and alienated from each other?
This could be the meaning of ezer k’negdo (Genesis 2:18) which Rav Yosef Dov Soloveitchik understands as Adam’s ‘discovery of a companion who even though as unique and singular as he, will master the art of communicating and with him form a community.’ (Lonely Man of Faith, p.26)
In Milton Steinberg’s words, real love is ‘to hold with open arms’.” 
Isn’t it natural to be attracted to another on a purely physical basis?
Yes — natural and unavoidable.
The Besht therefore teaches that we should remind ourselves: All beauty comes from the Divine Itself:
“If you happen to suddenly see a beautiful woman [or man; the teaching is applicable to either gender], think to yourself: ‘Where does this beauty come from? If she were dead, she would no longer look this way. So, it must come from the Divine Power [כח אלוה] within her. It gives her the quality of beauty and redness [i.e. health; vitality]. The root of beauty, therefore, is in the Divine Power [that is creating her and giving her life at this very moment]. Why, then, should I be drawn after a mere part? I am better off attaching myself to the ‘root and core of all worlds’ where all forms of beauty are to be found’. “ 
I’d add to this: If I, as a man, love a woman only for her physical beauty, then I’m only loving with that part of myself that is easily moved by physical beauty — i.e. the purely physical, instinctual part of me. The more of the entirety of the woman — as a person and a spiritual being — that I love, the more of myself I’m loving with.
Ultimately, it should be the Divine in me — expressed through the physical — that recognizes and loves the Divine in her.
And vice versa, of course.
“The more we love, the more does our love well…” 
 Pirkei Avot 5:16 or 19, depending on the edition.
 Hirsch, Rabbi Samson Raphael; Chapters of the Fathers; trans. into English by Gertrude Hirschler; Feldheim Publishers, © 1979; p. 89
 Goldin, Judah, trans. and ed.; The Living Talmud — The Wisdom of the Fathers; Mentor Books; © 1957 by Judah Goldin; p. 216
 Unterman, Isaac; Pirke Aboth; Twayne Publishers, © 1964 (reprinted by Bloch Pub.); p. 321
 Ps. 96:9
 Rabbi Avi Weiss, in a post on the value of nidah/separation between men and women during the menstrual cycle:
 Shochet, Rabbi Jacob Immanuel, trans.; Tzva’at Harivash — The [Ethical] Testament of Rabbi Israel Baal Shem; Kehot Publication Society, © 1998; p. 80 (paragraph # 90)
Note: The Besht is speaking here of recognizing the Divine in all things and at all times; not necessarily the question of love per se.
 Lichtenstein, Rabbi Morris; Jewish Science and Health; Jewish Science Publishing Co., © 1925; p. 134