The law requiring US citizens to "bear" a US passport when leaving and entering the US is the Immigration and Naturalization Act section 215, found at 8 USC § 1185, Travel control of citizens and aliens, subsection (b):
Except as otherwise provided by the President and subject to such limitations and exceptions as the President may authorize and prescribe, it shall be unlawful for any citizen of the United States to depart from or enter, or attempt to depart from or enter, the United States unless he bears a valid United States passport.
The limitations and exceptions generally comprise things like frequent-traveler program cards and trips to Canada (the latter exception is now only available to children who are traveling by land). I believe there's an exception for military personnel traveling on orders, too.
- What is the maximum prescribed penalty for violating these rules?
There is none. The law originally provided for a fairly stiff penalty, but it also originally applied in times of war. It also originally said "a valid passport" without specifying that it had to be a US passport. Over the years, the law was modified to its present form; the penalty provision was repealed in 1978.
There's an interesting history of this section at http://isaacbrocksociety.ca/2013/05/01/the-history-of-the-requirement-that-u-s-citizens-only-use-u-s-passports-to-enter-the-u-s/
- Are US immigration officials known for actually applying the penalty?
Well, no, because there isn't one. Furthermore, of course, US citizens have a legal right to enter the United States, so CBP immigration inspectors will admit them even when they don't have the proper documents, as long as the inspector is convinced that the applicant is a US citizen. It appears to be official policy that the entering citizen be advised of the necessity of having a passport and then admitted to the country.
I'm unaware of anything that documents current policy, but a 2006 manual for Customs and Border Protection Field Inspectors -- the officers who carry out immigration inspections -- reads thus:
12.5 United States Passport Waivers.
(a) General. Although primarily charged with the responsibility of determining citizenship, you are required to verify the validity of a United States passport when one is required by law. When an applicant fails to present a passport or presents an expired document, the immigration officer shall, if satisfied that the person is a United States citizen, advise the individual of the necessity of having a valid U.S. passport. Although technically you are waiving the passport requirement for the Department of State, no form need be completed. In addition, there is no fee collected by INS. (Paragraph (a) revised 10/21/98; IN99-02)
(Source: https://www.shusterman.com/pdf/cbpinspectorsfieldmanual.pdf. This manual was acquired through the Freedom of Information Act, and as far as I know there is no more recent version publicly available.)
Finally, all of this concerns US immigration inspectors and therefore assumes the traveler has reached the US border. No commercial carrier is likely to board a traveler who doesn't have appropriate documents, so nobody should expect to be able to fly to the US using (for example) a US birth certificate or US naturalization certificate. If a US citizen is abroad without a US passport, and doesn't want to or cannot get a new passport, it will be necessary to travel to the US by land or by private boat or aircraft.