What are rules or how do we find out the rules within Homer that determine the operation of the gen sets and the charging and discharging of the batteries?
HOMER operates the dispatchable power sources (generators, battery) in such a way as to minimize cost. Each hour, HOMER looks at the available dispatchable power sources and chooses to operate the source or the combination of sources that can produce the required amount of power and operating reserve most cheaply. For details, please refer to section 15.5.3 of our paper "Micropower System Modeling with HOMER". You can download that here. Section 15.5.4 explains the economic dispatch logic.
One important aspect of dispatch strategy in generator/battery/renewable systems is whether to charge the battery with the generator. We use the term cycle charging to describe the strategy whereby the generator charges the battery whenever it runs. Under the alternative strategy called load following, the generator does not charge the battery, but rather generates just enough power to serve the load. It's hard to know a priori which of the two strategies will be optimal in any particular application, so HOMER lets you try both to see which is optimal. The Generator Control Inputs window provides access to the relevant inputs.