NONE dare call it zoning, but the city of Houston - an urban monument to laissez-faire free enterprise - has begun to impose new controls that are having some of the same effects. Called ''development controls,'' they are slowly but surely altering the face of this city.

Alone among great American cities, Houston grew up without zoning, and with scarcely any planning. Some say this gave the oil capital the freedom to grow and boom and its developers the license to put up the daring and spectacular architectural palaces that are this city's hallmark. Others maintain that it has made the city something of a cluttered urban mess. Whatever the case, few believe that conventional zoning can now be imposed retroactively.

But the new rules are already having their effect: Motorists heading downtown along a handsome new freeway, Route 288, are struck by its lack of billboards, in contrast to the overwhelming signage clutter on the highways leading from the airports. On other roads, many billboards are coming down. New retail and commercial buildings are set back 25 feet from the street, allowing for landscaping and sidewalk cafes.

In many parts of town, massage parlors and other pornographic businesses have shut down under an ordinance limiting their concentration, and the exteriors of such businesses must be painted in achromatic tones - anywhere from black to white.

It is difficult to judge the full impact of the new rules because the slump in Houston's economy has brought development almost to a halt. But the rules represent the collective judgment of political and business leaders that they were ready to trade away some freedom for a measure of control in the interest of the commonweal. And there has been a growing realization that Houston's image has been damaged by commercial excesses, particularly the garish airport roads, the first vision most visitors get of ''Space City.''

''Since the economy has soured, a lot of business people have said visual appearance is very important to our future,'' said George Greanias, a city council member. ''We don't want to lose our individualism, but we have to take into account community interest. You cannot build up to the street line. You cannot put a massage parlor next to a church.''

Such talk would have been considered civic treason until Kathryn J. Whitmore became Mayor in 1981. She vowed to bring some order to Houston's urban development and began by replacing all but four of the 16 members of the City Planning Commission and recruited Efraim S. Garcia, a city planner who made a reputation in developing San Antonio's hugely successful Riverwalk.

Bringing order to Houston is not easy: Including its extraterritorial jurisdiction, the city has authority over 2,080 square miles of territory, an area larger than the state of Delaware. But zoning was never in the cards. ''If you put someone down in the middle of New York, Houston or Los Angeles, you couldn't tell that much difference,'' Mr. Garcia said. ''Zoning hasn't done that much to preserve neighborhood integrity.''

Indeed, bypassing conventional zoning altogether, Houston has emerged as a national pioneer in a new trend called ''performance zoning'' in some circles. Under this concept, developers are free to use the land for any purpose, but must comply with certain restrictions to minimize the impact on neighbors, such as through drainage requirements, setbacks, roadways and the like.

Is this, in effect, back-door zoning? ''It is not zoning because you can do what you want with the property,'' Mr. Greanias said, ''Traditional zoning is passe. But you must act in accordance with community needs.''

These are major elements of Houston's new controls:

* A billboard ordinance passed in 1980 that ends the licensing of new off-premise billboards and established 15 ''scenic districts'' in which signs taller than 12 feet must come down and billboards will be eliminated according to an amortization schedule required by state law: 21 1/2 years for steel mounted billboards and 17 years for wooden poles.