This paper aims to develop a driver-state estimation algorithm based on multi-modal information for various age and gender groups. A test bed was built under a simulated driving environment, and a total of 56 volunteers participated in a series of experiments that included involved states of drowsiness, distraction, and high workload. The algorithm to estimate the driver state was developed using a dynamic Bayesian network. The performance of the developed algorithm was verified and supplemented through vehicle, physiological, and image data obtained from the experiments. The algorithm showed a goodness of fit of 77.8% for correct detection rates greater than 0.7 and false alarm rates less than 0.3. The goodness of fit increased to 85.7% under the condition where the area of the receiver operating characteristic curve was more than 0.7.