Well that's entirely dependent on whether the Borg completely negate the death star weapons.
You only get a couple shots on any given frequency modulation of your energy weapons.
So as long as they have like.... 6 or 7 ships? They're good to go. After losing two or three ships. Just keep shooting until the death star goes boom.
If they hide inside a cargo ship and get tractor beamed into one of the bays, then I'd say it's only a matter of time before they control the entire death star.
It also depends on whether the Borg consider the death star a threat or worthy of assimilation. Given the accuracy of shooting of those on board, I highly suspect the ship's technology would be assimilated and then its inhabitants converted to drones.
The whole thing about being able to resist energy weapons is silly though, and the logic doesn't exrrapolate well at all when you have a laser the size of a small moon blasting at you. The Borg's only chance is to swarm the death star with too many ships too quickly and stay below its effective range.
The planet killer can't fire very frequently. Realistically, they just need to get within transporter range, send drones over, and start assimilating the ship. If they can avoid the first shot from the superlaser or just have a second ship, they're probably fine. Unless the Death Star has Vader on board.
Uhh, are we talking Rebels or Storm Troopers? I guess a battle of red shirts vs. storm troopers would be pretty epic. The red shirts would probably end up dying of cardiac arrest or something as lasers hit everything but them. You'd think they would have cloned someone with good aim for their elite troops.
The death star was shielded and armored against big ships, like capital ships (and maybe a Borg cube). It was defeated because they didn't bother with countermeasures against a small one-man fighter exploiting an Achilles' heel that only the rebellion knew about.
My guess would be not well. Their military technology doesn't seem to be anywhere near the equal of the Borg's. They would probably seek an alliance initially.