Not long after, the Confederate forces began the fight for the Red River. On April 8, 1864, at an intersection of two forest roads, three miles south of Mansfield, Louisiana, the Confederate forces of Major General Richard Taylor met Banks's forces. In two hours of solid fighting, Confederate troops pushed back the Union forces in a clear victory. In the night, the Federals retreated fourteen miles southeast to the village of Pleasant Hill. The two forces fought again on April 9. After a long day's fight, Bee was ordered back to the lines to set up pickets. He reported, after a final spasm of skirmishing, "the fire soon ceased and the time passed without alarm, but with noise, confusion, and moving of wagons, until midnight, when all was still." The next morning, the Federal troops were gone, moving south to Grand Ecore.

Of the routing and retreat of the Union soldiers, Ball wrote, "We gave the yanks a bad whip[p]ing." He further gloated: "The Feds throwed down all of their plunder and guns as they run.I got a blanket, and knapsack, oil cloth, and tent, and this paper and pencil that I am now using." By April 21, with the water level of the Red River falling and Banks's discouraged army in need of reinforcements, the Union army and navy began their retreat to Alexandria. In an attempt to surround Banks's army before they could move any further south, Taylor divided his forces in order to encircle the Federal army. Bee's men outraced Banks and took up position on the Cane River at Monett's Ferry. In a near disaster, Bee was outflanked and nearly surrounded. "He is the poorest excuse for a Gen that I ever saw," Ball later wrote.

Taylor's report on the encounter stated that Bee made four critical mistakes and that "he displayed great personal gallantry, but no generalship." The retreat of Bee's troops allowed Banks's army time to cross the Cane River and reach Alexandria, where the Union navy had been stranded due to the Red River's low water. There Banks set up his men in a defensive position around the city to give the navy time to dam the river in an attempt to make it past the rapids south of Alexandria. Two weeks later, the attempt succeeded, and Alexandria was evacuated and burned. By May 19, Banks's army had fought its way southeast to the Atchafalaya River. Upon its crossing, the Red River Campaign was considered closed.