it's afraid of your gigantic finger poking at it

Yes, there is something wrong. I would imagine that a big part of the problem is the size of your tank. You've only 1 gallon which is tiny. It means that unless you change the water very often the water will become quite toxic quite fast. I'm afraid your fish might be done for.

fish like to feed at night.

your tank is the problem, you need a minimum of 10 gal per goldfish, with 20 gal per goldfish a better option. The tiny fish you buy in pet shops are only 3 to 4 month old babies, less than a quarter their full grown size. . Remember that goldfish are a very large and very long lived fish, each of yours will eventually grow between 8 to 12 inches in length, (and could get as big as 18 inches long, and I have heard of rare times when a goldfish has grown to 32" in length.... remember, goldfish are river carp, and river carp are simply HUGE!) and once full grown will require a 20 - 30 gal tank PER FISH. . There is a common misconception that you can keep a goldfish in a bowl, because it is a small and short lived fish. Nothing could be farther from the truth. Goldfish live as many as 50 years, though 10 to 30 years is a more common lifespan. And a full grown goldfish can be well over a foot long. . Keep in mind when buying your goldfish, that the tiny 2 - 3 inch fish in the pet store is a baby about 3 months old. By the time they are 3 years old, your fish will be at least 8 inches long. . Why do goldfish so often die less than 4 inches long and under 2 years old? Because by keeping them in a small bowl or tiny tank, they have no room to grow, their growth become stauned, and they die at a young age as a result. . I recommend you do several things: . First off, get another tank, 10 gal at least, 20 is better. . Next, Put an overflow power filter on your tank IMMEDIATELY! Your fish is suffocating from lack of oxygen. . Third, add at least 1 airstone of no less than 4 inches long. . Check the PH level to be certain that it is between 7.2 to 7.6 as the wrong PH levels could be causing the problem as well. . Also test the ammonia and nitrate levels. If these are too high than they could be causing your fish to not get enough oxygen, even though you have a filter and airstone (bubbler). . The carbon in the filter should be replaced at least once a month, but some filters will require twice per month change. Also be sure to wash the filter sponge, using the old tank water (not tap water) to ensure that the "good bacteria" are not killed. (and if you are not using an over the top power filter, than you really should think about buying one, because this will greatly increase the oxygen level in your tank, and the carbon will help your fish as well.) . also, because goldfish are "messy" fish (meaning they produce a high rate of waste product in comparison to body size) it is imperative that you do a 1/4 to 1/3 water change each and every week, a 1/2 tank water change once per month, and a full tank cleaning (removal of fish to a mini-tank, while you completely remove all water, and hand wash the gravel, all filter parts, and plants, and decorations,) once every 6 to 8 months. Otherwise it would be the equivalent of you swimming in the city sewage system. . Remember, that goldfish are highly intelligent and social creatures. They MUST NEVER be kept alone. Buy a second goldfish or yours will pine away from loneliness.