You need to remove 'WHERE' from within your addCondition because addCondition will handle that for you. Also escape all your values with quotes but do not put quotes around your column names.
You should use CDbCriteria::compare or at least use addCondition with params for this to avoid SQL injection:
// Use the solution with CDbCriteria::addCondition
$criteria->addCondition(
"WHERE (jam_keluar > :jam_masuk OR jam_masuk < :jam_keluar)".
"OR (jam_masuk < :jam_keluar AND jam_keluar > :jam_keluar)",
array(
':jam_masuk' => $this->jam_masuk,
':jam_keluar' => $this->jam_keluar
)
);
// Or use the solution with CDbCriteria::compare
$criteria->compare('jam_keluar', '>'.$this->jam_masuk, false);
$criteria->compare('jam_masuk', '<'.$this->jam_keluar, false, 'OR');
$criteria->compare('jam_masuk', '<'.$this->jam_keluar, false, 'OR');
$criteria->compare('jam_keluar', '>'.$this->jam_keluar, false);