The dust has settled from the frantic first week of Openhack III, with its heavy traffic and mass of DoS attacks. The second week saw a lot more stability in the site and a bit more frustration from serious hackers still banging away as they rethought their approaches and tried to come up with unique methods to defeat Argus Systems Inc.s Pitbull before the Jan. 31 deadline.
Although Openhack III during the second week saw more uptime than in the first week, eWeek Labs did see a few crashes, especially in the Web server system, which crashed from brute-force attacks involving a few hundred thousand attempts to access the FTP server. However, although denial-of-service and standard attacks were the methods of choice during the first week, the second week saw more intriguing attempts to break the systems.
One of the more interesting developments occurred when a participant was able to gain the root user ID on the shell server system. In a typical Unix system, the game would be over at this point. But in a trusted operating system setup such as Pitbull, being the root user is much like being any other user: You still lack the privileges necessary to compromise the system.
The participant was able to create and modify files with root ID information but was unable to do so in areas where he lacked permission, including the root directory of the system.This prevented the participant from being able to achieve the successful hack of adding a file to the root directory.
Another participant was able to load a Perl script into the mail server on the Domain Name System and thus gain application-level access. This is a classic hacker tactic and one that was a key component of the successful hack in the last Openhack contest. But, again, without the needed permissions under the trusted operating system, this user was unable to do much on the system.
With the failure of these traditional types of attacks, talk on the Openhack III discussion list, at groups. yahoo.com/group/openhack3, turned to more advanced and nontraditional forms of attack.
The most promising tactic discussed was a direct attack on the operating system kernel. According to Argus engineers monitoring the event, one of the closest calls, so far, was an attempted kernel-level exploit.
However, there might not be enough time left to succeed in this most difficult type of exploit by Jan. 31. A kernel-level attack requires the deepest level of understanding of the operating system, and, even then, a flaw must be found for it to work.
Although this form of attack is difficult, it is not impossible. Several participants have already found potential flaws in the Solaris kernel that could lead to compromise of the Openhack shell system.